Summary

Two optional workflow_dispatch inputs on eval-canary.yml — the staging rails for Phase B and the pre-merge eval gap-closer from Phase D-1. Scheduled runs are completely unaffected (empty inputs resolve to today's exact behavior).

• rag_ranking_config — stages a RAG_RANKING_CONFIG JSON override for a single live eval run, the documented rollout path for tuned ranking weights (no code defaults touched; unset = production weights). A new validation step fails the run loudly on malformed JSON: resolveRankingConfig (src/lib/ranking-config.ts:183-191) silently falls back to production defaults on a parse error, which would turn a staged-weights eval into an accidental baseline and corrupt the before/after pair comparison. The step also echoes the active override into the log for auditability.
• ref — runs the eval from a branch/SHA (Phase D-1), closing the PR-template gap "CI cannot run the golden eval — needs live keys": ranking PRs can now gather live proof pre-merge via one dispatch.

First consumer (next step, separate dispatch): the tuned 3-class config from the offline tuner on the fresh A-PR-2 snapshot — document_lookup titleSectionRelevance→0.9, table_threshold clinicalEvidence→0.95, comparison hybridRelevance→0.95 (proxy rank quality 0.833→1.0) — validated live against the re-gated #53 baseline (mrr@10 ≈0.892) with targets mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions.

Verification

• npm run check:github-actions — pin check passed (no new actions; checkout pin unchanged)
• npm run check:ci-scope — passed
• Prettier clean
• No unit-test surface (workflow-only diff); behavior proof is the next dispatch itself: a run with an override logs RAG_RANKING_CONFIG override active: {...}, a malformed override fails at the validation step, and the weekly schedule continues running with empty inputs
• Verification not run: verify:pr-local — workflow-only change; the gates above are the repo's designated checks for .github/workflows/ diffs

Risk and rollout

• Risk: operational (workflow inputs). Two considered hazards: (1) silent-default corruption — mitigated by the loud validation step (the precise failure mode this PR exists to prevent); (2) ref runs branch code with the eval job's live secrets — bounded because workflow_dispatch requires repository write access, and this repo has a single trusted collaborator; the input only widens which commit an already-trusted dispatcher can evaluate. No new secrets, no permission changes, concurrency/issue-reporting behavior untouched.
• Rollback: plain revert — inputs disappear, schedule cadence unaffected throughout.
• Provider or production effects: none from this diff itself; it enables explicitly-dispatched staged evals (~$1–2 each, inside the user-raised ≤$20 ADDENDUM 4 cap, ~$1–2 spent).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — no answer-path change
• No patient-identifiable document workflow introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — env block unchanged apart from the pass-through override variable
• Service-role keys and private document access remain server-only — no new secret exposure
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — ranking defaults untouched; staged overrides are per-run and revert by omission
• Deployment classification/TGA SaMD impact: no production behavior change (eval tooling only)

Notes

• ADDENDUM 4 Phase B step 2 rails + Phase D-1 in one operational diff. On merge, the tuned-weights dispatch (task in flight) produces the live half of the pair.
• Ledger row appended (content commit f7f0b18).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Evaluation workflows now support optional ranking configuration overrides and branch or commit selection.
  • Invalid ranking configuration JSON is detected before evaluation runs.

• Documentation

  • Added ledger documentation covering the new workflow inputs, validation behavior, and trusted-access considerations.